On the Optimality Equation for Average Cost Markov Decision Processes and its Validity for Inventory Control

摘要

As is well known, average-cost optimality inequalities imply the existence ofstationary optimal policies for Markov Decision Processes with average costsper unit time, and these inequalities hold under broad natural conditions. Thispaper provides sufficient conditions for the validity of the average-costoptimality equation for an infinite state problem with weakly continuoustransition probabilities and with possibly unbounded one-step costs andnoncompact action sets. These conditions also imply the convergence ofsequences of discounted relative value functions to average-cost relative valuefunctions and the continuity of average-cost relative value functions. As shownin the paper, the classic periodic-review inventory control problem satisfiesthese conditions. Therefore, the optimality inequality holds in the form of anequality with a continuous average-cost relative value function for thisproblem. In addition, the $K$-convexity of discounted relative value functionsand their convergence to average-cost relative value functions, when thediscount factor increases to 1, imply the $K$-convexity of average-costrelative value functions. This implies that average-cost optimal $(s,S)$policies for the inventory control problem can be derived from the average-costoptimality equation.